Worst-case Analysis of Strategy Iteration and the Simplex Method

In this dissertation we study strategy iteration (also known as policy iteration) algorithms for solving Markov decision processes (MDPs) and twoplayer turn-based stochastic games (2TBSGs). MDPs provide a mathematical model for sequential decision making under uncertainty. They are widely used to model stochastic optimization problems in various areas ranging from operations research, machine learning, artificial intelligence, economics and game theory. The class of two-player turn-based stochastic games is a natural generalization of Markov decision processes that is obtained by introducing an adversary. 2TBSGs form an intriguing class of games whose status in many ways resembles that of linear programming 40 years ago. They can be solved efficiently with strategy iteration algorithms, resembling the simplex method for linear programming, but no polynomial time algorithm is known. Linear programming is an exceedingly important problem with numerous applications. The simplex method was introduced by Dantzig in 1947, and has since then been studied extensively. It can be shown that MDPs can be formulated as linear programs, thus, giving rise to the connection. Strategy iteration and simplex type algorithms are local search algorithms that repeatedly improve a current candidate solution. We say that the strategy iteration algorithm repeatedly performs improving switches, and that the simplex method repeatedly performs improving pivots. The strategy iteration algorithm and the simplex method are, in fact, paradigms for solving 2TBSGs, MDPs, and linear programs. To obtain concrete algorithms we must specify an improvement rule (pivoting rule). In this dissertation we mainly focus on three improvement rules: • Howard’s improvement rule: Update the current solution by simultaneously performing all improving switches. • RandomEdge: Update the current solution by performing a single uniformly random improving switch.